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METHOD AND APPARATUS FOR MODELING FILM GRAIN PATTERNS IN THE 

FREQUENCY DOMAIN 

CROSS-REFERENCE TO RELATED APPLICATIONS 

5 

This application claims priority under 35 U.S.C. 1 19(e) to U.S. Provisional Patent 
Application Serial No 60/498,945, filed August 29, 2003, the teachings of which are 
incorporated herein. 

1 0 TECHNICAL FIELD 

This invention relates to a technique for modeling film grain patterns in the frequency 
domain. 

1 5 BACKGROUND ART 

Motion picture film typically contains signal-dependent noise, often referred to as film 
grain, resulting from the process of exposure and development of the photographic film. Such 
noise yields a characteristic quasi-random pattern or texture, caused by the physical 

20 granularity of the photographic emulsion. Alternatively, signal-dependent noise can occur as 
result of subsequent editing of the images. The grain pattern can be simulated for video 
compression purposes. 

The rrU-T H.264 I MPEG-4 AVC video compression standard has accepted in its 
Fidelity Range Extensions Amendment the inclusion of a film grain SEI (Supplemental 

25 Enhancement Information) message. The film grain SEI message conveys a series of 

parameters allowing film grain simulation at the receiver. For the ITU-T H.264 I MPEG-4 
AVC compression standard, parameters in the SEI message can be specified according to two 
different models: the auto-regressive model and the frequency-filtering model. Both models 
allow characterizing the film grain pattern (size and shape), intensity and color correlation 

30 through different sets of parameters for different intensity levels. In particular, the frequency- 
filtering model characterizes the film grain pattern by specifying a set of cut frequencies that 
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define a 2D band-pass filter in the frequency domain. Note that ITU-T H.264 I MPEG-4 AVC 
only standardizes the syntax necessary to transmit the cut frequencies but does not provide a 
method for computing them for a video sequence with film grain. 

Thus, there exists a need for a technique allowing the automatic modeling of the film 
5 grain pattern in the frequency domain as specified by the frequency-filtering model in ITU-T 
H.264 I MPEG-4 AVC compression standard. Results for this technique could be used either 
for automatic film grain modeling applications or as the initialization step for a film grain 
assisted-modeling process. 



1 0 BRIEF SUMMARY OF THE INVENTION 



Briefly, in accordance with a preferred embodiment, there is provided a method for 
modeling (i.e., characterizing) film grain patterns in the frequency domain. The method 
comprises the steps of (1) transforming an set of homogeneous film grain samples received as 

15 an input to the process to the frequency domain, thereby yielding a set of transform 

coefficients having a particular pattern; (2) analyzing the pattern created by the transformed 
coefficients; and (3) estimating the cut frequencies of a 2D frequency filter that can 
effectively simulate the pattern of transform coefficients by filtering random noise. The cut 
frequencies established by this method can be conveyed in an SEI message in accordance with 

20 the rrU-T H.264 I MPEG-4 AVC standard allowing film grain simulation and reinsertion at a 
decoder. 



BRIEF DESCRIPTION OF THE DRAWINGS 



25 FIGURE 1 depicts in flow chart form the steps of a method for characterizing film 

grain patterns in accordance with the present principles; and 

FIGURE 2 depicts in flow chart form a variation of film grain characterization method 
of HG. 1. 



30 
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DETAILED DESCRIPTION 

FIGURE 1 depicts in flow chart form the steps of a method in accordance with present 
principles for modeling a film grain pattern in the frequency domain upon receipt of a series 
of film grain samples representing a homogeneous film grain pattern. As discussed in greater 
detail hereinafter, the method of the present principles parameterizes the pattern of the input 
samples by analyzing the size and shape of the structures forming the grain. Because grain 
can form differently depending on film exposure, homogeneous film grain samples are 
typically those associated with simdlar luminance values measured on the film image. Film 
grain samples at the input of the process could be any group (or groups) of neighboring pixels 
that retains information about the film grain size and shape. In the illustrated embodiment, we 
will assume for simplicity that the film grain samples are arranged in square blocks of NxN 
pixels with a particular transform implementation based on a DCT of squared blocks of NxN 
pixels, although other transforms, such as a Fast Fourier Transform work equally as well. 

The method of the present principles assumes that modeling of the film grain present 
in Igrain[x, y, c] occurs in accordance with the relationship: 

Igrainl^j y> ^3 ~ Iwithout grain 

[X, y, c] + G[x, y, c] (1) 

where G[ x, y, c] represents the simulated grain at pixel coordinates (x, y) for color 
component c. G[ x, y, c ] is computed as: 

G[x,y,c]= p*Q[x,y,c]+u*G[x,y,c-l] (2) 

where the parameter p is the standard deviation of the random noise and the parameter u 
models the cross-color correlation among different color components. More particularly, the 
term Q[ c ] comprises a two-dimensional random field generated by filtering blocks b of N x 
M random values, which have been generated with a normalized Gaussian distribution N(0,1). 
In a particular embodiment, the band-pass filtering of blocks b can be performed in the 
frequency domain by the following three steps: 
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Step 1: Transform 

B = DCT.N X M( b) 

5 Step 2: Frequency filtering 

for( y=0; y<N; y++) 

for( x= 0; x<M; x++) 
if( (x<LOW_HF && y<LOW_VF) 11 
10 x>HIGH„HF 11 y>HIGH„VF) 

B[ x,y] = 0; 

where LOW_HF and LOW_VF are the low Horizontal and Vertical cut frequencies, 
respectively, and HIGH.HF and HIGH^VF are the high Horizontal and Vertical cut 
15 frequencies, respectively. The cut frequencies define the boundaries between preserved and 
filtered coefficients when a film grain image is mapped in the frequency domain and serve to 
characterize the size of the grain. 

Step 3: Inverse Transform 

20 

b' = IDCT_N X M( B) 

Finally, Q[ c ] is formed by combining the filtered blocks b' into a composite image. 
Low pass filtering of the block transitions will reduce possible "blockiness." Although M and 
25 N could take any value, in practice squared blocks of 1 6x 1 6, 8x8 or 4x4 pixels work best. 
Note also that other transforms, such as the Fast Fourier Transform (FFT), could replace the 
DCT process in Steps 1 and 3. 

By these principles, modeling the film grain patterns is equivalent to extracting the cut 
30 frequencies LOW^HF, LOW^VF, HIGH.HF and HIGH^VF that characterize the band-pass 
filter in the frequency domain. 
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The method of the present principles commences upon execution of step 101, in which 
each block of NxN pixels undergoes a Discrete Cosine Transform, with subsequent storage of 
the resulting arrays of NxN coefficients during step 102. During step 103, a check occurs to 
decide whether a need exists for more blocks with film grain samples in order to obtain more 
5 coefficients for storage. Ordinarily, all blocks of film grain samples available at the input 
undergo a transform. However, to reduce memory requirements or computational load, 
processing could stop after a certain number of blocks have undergone a transform. 
Following storage of a sufficient number of transformed blocks, step 104 occurs, whereupon a 
mean block (Bmean) is computed by averaging the coefficients from all the stored blocks. 
10 Assuming K as the number of stored blocks, the averaging process for the coefficient at 
position [x,y] can be formulated as follows: 



1 

B .naan U» 3^] = ^ H ^ ' (3) 

15 Next, steps 105 and 106 occur typically in parallel. During step 105, a horizontal mean vector 
Bh is computed by averaging the N frequency coefficients of each row of Bmean in accordance 
with the relationship: 



20 



B „ [ y ] = — X B [n, y] (4) 



In a particular embodiment, it is possible to avoid the influence of the DC coefficient on the 
average of the first row with the relationship: 



25 

During step 106, the vertical mean vector is computed by averaging the N frequency 
coefficients of each column of Bmean in accordance with the relationship: 
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BvW = -^i;B„,„„[x.n] (5) 

In a particular embodiment, it is possible to avoid the influence of the DC coefficient on the 
average of the first column vs^ith the relationship: 

5 

Bv[0] = -^X;B--[0,n] 

From the frequency vectors, selection of the horizontal and vertical cut frequencies 
occurs during steps 107 and 108, respectively, to estimate the film grain size. As seen in FIG» 
10 1, steps 107 and 108 typically occur in parallel. Horizontal cut-frequency selection during 

step 107 occurs in the following manner. First, the components in the horizontal mean vector 
undergo low-pass filtering to avoid spurious peaks. In the illustrated embodiment, such low 
pass filtering of the horizontal mean vector occurs by convolving the mean vector with a filter 
of impulse response h[n] in accordance with the relationship; 

15 

B^;,[n] = Xi»HWh['»-*] = (BH (6) 



For example, a 3-tap linear filter with coefficients wq, Wi, and W2 could be applied to each 
coefficient in accordance with the relationship: 

20 

B'^[n] = Wo B^[n-l] + w, -3^^+ B^ln + l], 0<;x<iV-l (7) 

Observe that in order to apply the filtering on the edges of the mean vector B it is necessary to 
pad the original mean vector so that the samples for n < 0 and n > N-1 are defined. 

25 

Next, the mean value of B h is computed by averaging its components in accordance 
with the relationship: 



wo 2005/027045 



PCT/US2004/010789 



-7- 

^'«=-^ZB'«W (8) 

^ 11=0 

Thereafter, the vector B h is represented as a curve, and its intersection points with the average 
value jB'^ are computed. If a single intersection point is found, the index n of the closest 

5 component in B'h is chosen as the value of the horizontal high cut frequency; the horizontal 
low cut frequency is assumed to be 0. If two intersection points are found, the indexes of the 
closest components are found for each one. The lowest value will correspond to the low 
horizontal cut frequency whereas the highest value will correspond to the high horizontal cut 
frequency. If more than two' intersection points are found, no spatial correlation is detected. 

10 The horizontal low cut frequency is assumed to be 0, and the horizontal high cut frequency is 
assumed to be N-1, indicating to the film grain simulation function that no frequency filtering 
is required to imitate the original grain. 

The same procedure described for selecting the horizontal cut frequency occurs during 
step 108 to select the vertical cut frequency using the vertical frequency vector By- At the 

15 completion of steps 107 and 108, the method of FIG. 1 yields four cut frequencies (LOW^HF, 
HIGH_HF, LOW_VF, HIGH^VF) that characterize both the size and the elongation of the 
grain. Elongated grain occurs when LOW^HF ^ LOW_VF and/or HIGH^HF HIGH^VF. 

FIGURE 2 illustrates an alternative grain modeling method, where it is possible to 
constrain the grain to circular shapes. This implies that the horizontal and vertical cut 

20 frequencies remain the same. The method of FIG. 2 contains many steps in common with the 
method of FIG. 1. Therefore, like reference numerals have been used in FIG. 2 as in FIG. 1 to 
describe like steps. The method of FIG. 2 differs frorri that of FIG. 1 in that , the vertical and 
horizontal frequency vectors (Bh and By) are averaged during step 109 of FIG. 2 to create 
single frequency vector (B). Then, the same procedure is performed during steps 107 and 108 

25 in FIG. 2 to estimate low and high cut frequencies as is performed during steps 107 and 1 08 of 
FIG. 1.. 

The foregoing describes a technique for modeling a film grain pattern in the frequency 
domain. 



wo 2005/027045 



PCT/US2004/010789 



-8- 

CLAIMS 

1. A method for automatically modeling film grain patterns, comprising the steps 

of: 

transforming a set of falm grain samples to the frequency domain; 
5 storing each set of coefficients resulting from such transform, the coefficients forming 

a pattern; 

analyzing the pattern created by the transform coefficients; and 

estimating the cut frequencies of a 2D band-pass filler that can effectively simulate the 
pattern of transform coefficients by filtering random noise in a firequency domain. 

10 

2. The method according to claim 1 further comprising the step of transmitting at 
least one cut frequency in a Supplemental Enhancement Information message . 

3. The method according to claim 1 wherein the film grain samples are processed 
1 5 in blocks of NxN pixels. 

4. The method according to claim 3 wherein the step of analyzing the pattern 
created by the transform coefficients further comprises the steps of: 

computing a mean block of NxN transform coefficients by averaging the transform 
20 coefficients from all the stored blocks; 

defining horizontal and vertical mean vectors of N components each by averaging the 
mean block of NxN coefficients along rows and columns, respectively, of each transformed 
block; 

representing the horizontal and vertical mean vectors as separate curves; and 
25 establishing horizontal and vertical cut frequencies from the curves represented by the 

horizontal and vertical mean vectors, respectively. 

5. The method according to claim 4 further comprising the step of low pass 
filtering at least one mean vector. 

30 
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6. ' The method according to claim 4 wherein the at least one cut frequency is 
established from an intersection point in the curve representing the mean vector. 

7. The method according to claim 4 wherein each of a low and a high cut 

5 frequency is established from a first and second intersection points in the curve representing 
the mean vector. 

8. The method according to claim 3 wherein the step of analyzing the pattern 
created by the transform coefficients further comprises the steps of: 

computing a mean block of NxN transform coefficients by averaging the transform 
coefficients from all the stored blocks; 

defining horizontal and vertical mean vectors of N components each by averaging the 
mean block of NxN transform coefficients along rows and columns, respectively, of each 
transformed block; 

averaging the horizontal and vertical mean vectors into a single mean vector; 
representing the mean vectors as a curve; and 

establishing horizontal and vertical cut frequencies from the curve represented by the 
mean vector. 

20 9. The method according to claim 8 further comprising the step of low pass 

filtering the mean vector. 

10. The method according to claim 8 wherein at least one cut frequency is 
established from an intersection point in the curve representing the mean vector. 

25 

11. The method according to claim 8 wherein each of a low and a high cut 
frequency is established from a first and second intersection points in the curve representing 
the mean vector. 

30 
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12. A method for automatically modeling film grain patterns, comprising the steps 



of: 



receiving a set of film grain samples 

performing a transform on the set of film grain samples to the frequency domain 
storing each set of coefficients resulting from such transform, the coefficients forming 
a pattern; 

analyzing the pattern created by the transform coefficients; and 

estimating the cut frequencies of a 2D band-pass filter that can effectively simulate the 
pattern of transform coefficients by filtering random noise in a frequency domain. 
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